An electronic imaging system depends on an electronic image sensor to create an electronic representation of a visual image. Examples of such electronic image sensors include charge coupled device (CCD) image sensors and active pixel sensor (APS) devices (APS devices are often referred to as CMOS sensors because of the ability to fabricate them in a Complementary Metal Oxide Semiconductor process). A sensor consists of a two-dimensional array of individual picture element sensors, or pixels. Regardless of electronic technology employed, e.g., CCD or CMOS, the pixel acts as a bucket in which photoelectrons are accumulated in direct proportion to amount of light that strikes the pixel. Photoelectrons are electrons that are created due to the interaction of light with the pixel and, therefore, represent the signal being detected by the pixel. Thermal electrons are created by the thermal conditions of the device and are generally not related to the light being sensed by the pixel. However, thermal electrons will coexist with photoelectrons within a pixel and are indistinguishable from photoelectrons. Thermal electrons represent a major source of noise in the response of the pixel.
In most commercially available sensors today, the maximum ratio of signal to noise for a pixel is about 100:1 which represents the maximum dynamic range of the pixel. Since the human visual system, at any given moment, is operating with a dynamic range of about 100:1, there is a good correspondence between the human visual system and the image capture capability of the sensor. However, scenes in nature often consist of visual information over a dynamic range that is much greater than 100:1. The human visual system is constantly adapting its brightness sensitivity so that the most visually important information stays within its 100:1 dynamic range capability. However, most electronic image sensors have no such real-time adjustment capability. It is up to the camera's exposure adjustment system to properly regulate the amount of light falling on the sensor. If the exposure adjustment system makes an error, and selects the wrong portion of the scene to capture within the dynamic range of the sensor, the resulting image has either shadows that are too dark or highlights that are too light. In the case where the important parts of a scene consist of visual information over a dynamic range that is greater than 100:1, some of the important parts of a scene will be clipped regardless of the regulation by the exposure adjustment system.
If the dynamic range of the pixel could be increased from 100:1, more scene information could be recorded at capture time and subsequent image processing could properly create an image with the desired rendering. However, the current industry trends in sensor manufacturing are to make pixels smaller and sensors cheaper. The smaller the pixel size, the fewer total photoelectrons it can accumulate. Since the number of thermal electrons accumulated stays roughly the same as the pixel shrinks in size, the overall result is that smaller pixels have smaller dynamic ranges.
U.S. Pat. No. 6,040,858 issued Mar. 21, 2000 to Ikeda provides a complete description of the problem of the limited dynamic range of electronic image sensors. In addition, Ikeda describes methods of extending the dynamic range of an electronic image sensor by capturing multiple image signals with different exposures. These multiple signals are combined by using thresholds that determine which signal is of higher quality at each position in the image signal to form an image signal having extended dynamic range. Ikeda improves upon these methods by describing a method by which these thresholds are determined for each color. Using Ikeda's method, a very high dynamic range can be achieved by combining many images. However, the multiple images must be properly aligned because moving objects in the scene that change location from one image capture to the next introduce artifacts in the final image.
Another way of producing an image with an extended dynamic range is by employing an electronic image sensor that has high sensitivity photosites interspersed with low sensitivity photosites (U.S. Pat. No. 6,943,831 issued Sep. 13, 2005). The difference in sensitivity between the high sensitivity photosites and the low sensitivity photosites is achieved by applying different gains to the two types of photosites. The maximum gain in a conventional electronic image sensor is typically selected to be the highest gain that can be applied while still producing an image that is pleasing and without too much noise. This maximum gain is applied, in the case of the apparatus described in U.S. Pat. No. 6,943,831, to the high sensitivity photosites and a gain that is lower than the maximum gain is applied to the low sensitivity photosites. After a capture, the pixel values generated by the high sensitivity photosites in very dark areas of the image are used to replace the pixel values generated by the low sensitivity photosites in the same areas, and the pixel values generated by the low sensitivity photosites in very light areas of the image are used to replace the pixel values generated by the high sensitivity photosites in the same areas, to form an image with an extended dynamic range. This method requires only one image capture to produce an extended dynamic range image. Therefore, scene object motion does not pose a problem when combining high pixel values with low pixel values. However, since the gain that is applied to the high sensitivity photosites is the same gain that would be applied in a conventional electronic image sensor, it is necessary to employ slow shutter speeds if extremely dark areas of a scene are to be imaged within the extended dynamic range. Images captured with slow shutter speeds tend to be blurrier than those captured with fast shutter speeds.
U.S. Pat. No. 6,864,916 issued Mar. 8, 2005 to Nayar describes another method that extends the dynamic range of an electronic image sensor. Nayar's method includes the utilization of an optical mask with spatially-varying transmittance, thereby forcing the effective response of each photosite in an electronic image sensor to change according to the amount of light impinging upon each photosite. Following Nayar's approach, the photosites of an electronic image sensor are most sensitive when sensing light from a very dark portion of a scene and the photosites of an electronic image sensor are least sensitive when sensing light from a very light portion of a scene. Nayar's approach does mimic the brightness adaptation property of the human eye. However, Nayar's method also requires using both slow shutter speeds, if extremely dark areas of a scene are to be imaged within the extended dynamic range, and a complicated and costly optical mask that cannot be used without first modifying the hardware of current image capture systems.
Thus, there exists a need for generating extended dynamic range images by utilizing conventional electronic image sensors with a reduced amount of images while imaging very dark areas of a scene within the extended dynamic range without using slow shutter speeds.